An Introduction to Support Vector Machines: A Review

In the preface of the book, Cristianini and Shawe-Taylor state that their intention is to present an organic, integrated introduction to support vector machines (SVMs). The authors believe that SVMs are a topic now sufficiently mature that it should be viewed as its own subfield of machine learning. SVMs, first introduced by Vladimir Vapnik, are a type of linear learning machines much like the famous perceptron algorithm and, thus, function to classify input patterns by first being trained on labeled data sets (supervised learning). However, SVMs represent a significant enhancement in function over perceptrons. The power of SVMs lies in their use of nonlinear kernel functions that implicitly map input into high-dimensional feature spaces. In the high-dimensional feature spaces, linear classifications are possible; they become nonlinear in the transformation back to the original input space. Thus, although SVMs are linear learning machines with respect to the high-dimensional feature spaces, they are in effect nonlinear classifiers. The authors review and synthesize a wide range of materials, including the dual representation characteristic of linear learning machines, feature spaces, learning theory, generalization theory, and optimization theory, that are necessary for a comprehensive introduction to SVMs. The topics are introduced in an iterative and problem-triggered manner: Problems are presented, followed by concepts the last section of chapter 2, the dual representation of linear learning machines is introduced. Dual representation is one of the crucial concepts in developing SVMs. The limited computational power of linear learning machines leads to the topic of the third chapter, “Kernel-Induced Feature Spaces.” To increase the computational power of the linear learning machines, nonlinear mappings can be used to transform the data into a high-dimensional feature space in which a linear learning methodology is then applied. Kernel functions can implicitly combine these two steps (nonlinear mapping and linear learning) into one step in constructing a nonlinear learning machine. A linearly inseparable problem can become linearly separable in a higher-dimensional feature space. As a consequence of the dual representation of linear learning machines, the dimension of the feature space need not affect the computation because only the inner product is computed by evaluating the kernel function. The use of kernel functions is an attractive computational shortcut. The use of kernel functions to construct nonlinear learning machines greatly increases the expressive power of learning machines and retains the underlying linearity that ensures the tractability of learning. However, the increased flexibility increases the risk of overfitting, which can lead to bad generalization performance. Chapter 4, “Generalization Theory,” introduces the theory of Vapnik and Chervonenkis (VC) to control the increased flexibility of kernel-induced feature space and lead to good generalization. Loosely speaking, the most important result of VC theory is that the upper bound of the generalization risk for a learning machine is controlled by the empirical risk and the VC dimension, which is fixed for a hypothesis space of the learning machine. The theory presented in chapter 4 shows that the learning machine with the lowest upper bound of generalization risk is achieved by selecting a machine that minimizes the empirical risk. This discussion sets An Introduction to Support Vector Machines